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ABSTRACT 

Research for Better Schools and a Maryland county 
school district developed a school improvement program to examine the 
relationship between the program and subsequent improvement* The 
program involved a self-assessment survey, choosing goals for the 
following year, assigning a committee of practitioners to lead the 
program, executing the plan created by the committee, and completing 
the self-assessment survey at the end of the school yoar to analyze 
the amount of inqprovement . The Dimensions of Excellence Scales (DoES) 
was employed to assess program effects and the School Survey was 
employed to measure program implementation. The study reveals that 
every school except one significantly improved; schools that achieved 
high faculty conqpletion rates on the DoES made only moderate school 
improvement; no relationship seemed to exist between participa-cion in 
the program and the amount of improvement; improvement was affiliated 
with the original pretest status; and schools with better pretest 
scores attracted more faculty personnel to the program* Essentially, 
no consistent relationship was evident between involvement in the 
program and school improvement. Future school improvement efforts 
should consider the context of the school. Sixteen references and 7 
tables are included. (RG) 
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Over the past decade, countless studies have been conducted that 
examine the effectiveness of elementary schools. The majority of these 
studies has attempted to identify characteristics of effective schools 
(Edmonds, 1979; Wilson & Corcoran, 1987), differentiate effective from 
ineffective schools (Austin, 1978; Brookover & Lezotte, 1979; Edmonds & 
Frederiksen, 197S), or discuss the validity of these efforts (Brookover, 
1987; Purkey & Smith, 1983; Stedman, 1987). Very little of the school 
effectiveness literature has reported on the systematic efforts of schools 
to improve their status on some continuum (from ineffective to effective) 
associated with effective school practices and even fewer to relate program 
implementation indicators with outcome measures (Mile^, Farrar, & Neufeld, 
1983; Purkey & Smith, 1983). Although there have been numerous school 
district initiatives aimed at such a purpose (Miles, et al, 1983), these 
have typically not been reported in the research literature. The current 
paper r6po:.\;s on one such effort to investigate the relationship between 
participation in a school improvement program and subsequent school 
Improvement. 



Background 

In late 1984, a Maryland county school district and Research for Better 
Schoolr (RBS) agreed to work together to develop a comprehensive school 
improvement program following an action research model. Participating 
schools in the district would complete the following steps as part of this 
program: 



• complete a self-assessment survey on eight dimensions associated 
with effective elementary school practices 

• review the resulting profile of the school on the eight 
dimensions 

9 select a challenge dimension on which to focus the school's work for 
the coming year 

• appoint a committee of administrators and classroom teachers to 
lead each school's effort to improve its performance on the 
challenge dimension 

• carry out a plan developed by the committee to improve the school's 
performance on the challenge dimension 

• complete the self-assessment survey at the end of the school year 
to determine the extent of the school's improvement on the 
challenge dimension. 

The first two activities were structured by the program to occur at general 

school-wide faculty meetings; the remaining activities were not structured 

formally by the program and so allowed individual schools to organize the 

program to meet their own needs- Individual schools could choose to focus 

their work on any of eight dimensions — school climate, leadership, teacher 

behavior, curriculum, monitoring and assessment, student behavior and 

discipline, staff development, and parent involvement. However, schools 

typically elected to work on one of their lower scoring dimensions. The 

program relied heavily on the involvement of classroom teachers in the 

selection of the challenge dimension as well as the development and conduct 

of the school's improvement plan to address the challenge dimension. This 

program is described elsewhere (RBS, 1987) and so will not be described in 

more detail here. 



Methodology 

As noted above, this paper examines the progress of individual schools 
involved in a school improvement effort • More specifically, this paper 
attempts to relate program implementatiox* indicators to changes in the six 
schools* status on their challenge dimension. Three of the schools (Schools 
A, B, and C) volunteered to participate in the program during its first 
year; the other three schools (Schools D, E, and F) joined the program in 
its second year. This paper examines data gathered only during the first 
year of the schools' participation in the program. 

A variety of strategies was used to measure program implementation and 
effects in the six schools. Of r.ost interest to this paper are school 
administrator and faculty responses to two instruments--the Dimensions of 
Excellence Scales (DoES) and .the School Survey . 

The Does was used to measure program effects. Specifically, the DoES 
measures an individual school's status on eight dimensions associated with 
effective elementary school practices. In completing the survey, 
respondents are asked to rate their school (using a 5-point scale) on a 
series of items (16-33 items per dimension) associated with effective 
practices on each of the eight dimensions. Fcr example, some of the items 
used to assess the school's performance on the school climate dimension 
include : 

• the school has an orderly, business-like atmosphere 

• the school motivates students to learn 

• the principal has a high level of visibility. 



Respondent ratings are aggregated by dimension and reported as raw 
scores and scale scores (based on the district mean) . Individual school 
profiles are returned to the schools for review and use in focusing their 
work for the coming year or in assessing the outcomes of their work at the 
end of the school year. 

The Does was completed at the beginning of participation in the program 
and then at the end of each year of participation • In the case of the six 
schools, the DoES was completed by the principal, classroom teachers, and 
instructional aides assigned to each school. Specific numbers completing 
the DoES are reported below. 

Table 1 

Numbers of School Personnel Completing the DoES 



School 


Pretest 


Posttest 


A 


21 


27 


B 


36 


40 


C 


38 


33 


D 


23 


27 


E 


52 


54 


F 


65 


69 



Differences in the numbers from pretest to posttest represent changes in 
staff assignments and absenteeism on days when the DoES was completed. In 
order to protect the anonymity of individual staff members, there was no 
attempt to match responses from one administration to the next. Instead, a 
school's profile reflects the aggregated responses of the school 
administrators and faculty who completed the DoES at a given time. 

The School Survey was used to assess program implementation. It was 
developed to document school personnel's involvement and their ratings of 



the conduct and impact of the program in their school. The survey is 
divided into a number of sections. In addition to collecting basic 
demographic data (i.e., name of school and dimension on which the school 
focused its work) , the survey sections collect information on the following 
broad areas: 

# participation in various generic program activities during the 
school year 

# provision of various supports within the school building necessary 
for effective implementation of the program 

# importance and value of school's effort related to the challenge 
dimension. 

The items are generally statements on which respondents are asked to report 
on their involvement (i.e., participation in progreun activities) or to rate 
the performance of their school (i.e., sharing of program leadership, 
relevance of strategies). Respondent ratings are aggregated for each of the 
survey items by school and means and standard deviations calculated and 
reported • 

The School Survey was completed by school administrators and faculty at 
the end of each year of participation in the program. However, somewhat 
different samples of school personnel completed the survey. In Spring, 1986 
the survey was completed at the three schools that joined the program during 
its first year (Schools A, B, and C) by samples of school personnel, 
generally representing the principal and approximately one-third to one-half 
of the faculty. In Spring, 1987 the survey was distributed to all personnel 
assigned to the second wave of schools (Schools D, E, and F). Table 2 
summarizes the number and percent of school personnel who completed the 
survey by school building. 



Table 2 

Number and Percent of School Personnel Completing the School Survey 



School 


Number 


Percent 


A 


16 


55.2 


B 


13 


32.5 


C 


14 


36.8 


D 


17 


62.9 


E 


34 


63.0 


F 


62 


89.9 



No systematic bias was found in thi> sample of faculty completing the survey 
during the first year. 

Analysis and Results 
Independent ''f-tests were conducted to determine if the six schools 
made significant improvements on their respective challenge dimensions. All 
but one school (School C) made statistically significant gains. 
Interestingly, the magnitude of the changes was not related to volunteer/ 
non-volunteer status (see Table 3). 

Table 3 



Changes in DoES Challenge Dimension Scores 



School 


ChallenKe 


Pretest 


Posttest 


Gain 


t 


P 


A 


Climate 


4.06 


4.38 


.32 


2.91 


*.05 


B 


Behavior & 


4.06 


4.46 


.40 


3.08 


«.05 


C 


Discipline 

Parent 

Involvement 


3.50 


3.59 


.09 


.60 


NS 


D 


Behavior & 


4.22 


4.82 


.60 


5.01 


<.05 


E 


Discipline 
Qlimate 


3.79 


3.92 


.13 


1.30 


<.10 


F 


Climate 


3.28 


3.69 


.41 


4.07 


<.05 



6 

8 



The two largest gains were obtained by Schools D and F (,60 and ,41 
respectively) which registered the highest and lowest starting points on 
their respective challenge dimensions. The two smallest gains were achieved 
by two of the schools with low pretest scores (Schools E and C, .13 and ,09 
respectively). These data support the absence of a ceiling effect. 

The next set of analyses investigated the relationship between 
performance gains on the DoES challenge dimension and participation in 
program activities by school personnel (obtained from responses to the 
School Survey ) . It was hypothesized that the magnitude of school 
improvement would correlate positively with the percentage of faculty 
participating in program activities. In order to test this hypothesis, 
Spearman rank order correlations (Siegel, 1956) were calculated. As 
reported in Table 4, only one significant rank correlation was found. 
Completion of the DoES by school personnel (a program-structured activity) 
was negatively correlated with gains on the DoES challenge dimension. In 
other words, schools that achieved high faculty completion rates on the DoES 
made only modest gains on the school improvement measure. No consistent 
relationship was found between participation in the other six program 
activities and the magnitude of school improvement (as measured by gains on 
the DoES challenge dimension) . 

In order to investigate the relationship between school implementation 
of the program and subsequent improvement on the challenge dimension more 
fully, the six schools were divided into two groups (i.e., top three, bottom 
three) based , on their initial scores on ♦'he DoES challenge dimension 
selected by t! school* These rankings are presented in Table 5 along with 



Table U 

Relationship Between Program Pavtlclpatlon and Subsequent School Improvement Gain 

Percent of School Personnel Participating in Program Activities 

Influenced 



School 


Completed 

Does 


Discussed 
DoES results 


Influenced 
selection of 
challenge 


Analyzed/ 

defined 

challenge 


strategies to 

address 

challenge 


Attended 
staff 

development 


Implemented 
strategies 


A 

A 






L'^ ft 


L'k ft 




fti ^ 


fti ^ 


B 


84.6 


84.6 


46.2 


53.8 


38.5 


76.9 


92.3 


C 


100.0 


100.0 


35.7 


35.7 


14.3 


64.3 


64 3 


D 


76.5 


64.7 


29.4 


17.6 


23.5 


52.9 


58.8 


E 


97.1 


61.8 


14.7 


17.6 


17.6 


32.4 


44.1 


F 


88.9 


60.3 


7.9 


15.9 


7.9 


17.5 


38.1 


s 

r 


-.94 


-.49 


-.49 


-.03 


-.09 


-.26 


-.49 


P 


<.05 


NS 


NS 


NS 


NS 


NS 


NS 



the gains each school achieved during the fi-rst year of their participation 
in the program. 

Table 5 

Rankings of Six Schools Based on Pretest DoES Scores 



Rank 


Pretest 


Gain 


1 


School D (4.22) 


.60 


2.5 


School B (A. 06) 


.40 


2.5 


School A (A. 06) 


.32 


A 


School E (3.79) 


.13 


5 


School C (3.50) 


.09 


6 


School F (3.28) 


.41 



A Mann-Whitney U test (Siegel, 1956) was conducted to determine if there was 
any significant difference between the gains achieved by the two groups. 
The results indicated that improvement was positively associated with 
initial status on the pretest measure (p<.05)^ In other words- the top 
group of schools was found to achieve greater gains than the bottom group of 
schools. It's also interesting to note that only minor changes occurred in 
the relative rankings of the six s-hools; none of the six schools moved 
from either the high or low group to its counterpart. 

Given this finding, analyses to determine the relationship between 
program j,articipat,ion and performance gains on the challenge dimension W5>re 
repeated to see /x there were any significant differences between the two 
groups. Mann-Whitney U tests (Siegel. 1956) were used to test for 
differences. , As reported in Table 6, the top group surpassed the bottom 
group on all six school-structured program activities. In general, they 



Table 6 

Relationship Between Group Membership and Participation in Program Activities 



Percent of School Personnel Participating in Program Activities 



School 



Completed 

Does 



Discussed 
DoES results 



Influenced 
selection of 
challenge 



Analyzed / 

defined 

challenge 



Influenced 
strategies to 
address 
challenge 



Attended 
staff 

development 



Implemented 
strategies 



D 
B 
A 



76.5 
84*6 
93.8 



64.7 
84.6 
75.0 



29.4 
46.2 
43.8 



17.6 
53.8 
43.8 



23.5 
38.5 
50.0 



52.9 
76.9 
81.3 



58.8 
92.3 
81.3 



E 
F 
C 



97.1 
88.9 
100.0 



61.8 
60.3 
100.0 



14.7 
7.9 
35.7 



17.6 
15.9 
35.7 



17.6 
7.9 
14.3 



32.4 
17.5 
64.3 



44.1 
38.1 
64.3 



NS 



NS 



^.10 



< .10 



<.05 



-.10 



<.10 



Probabilities are based on one- tailed Mann-Whitney U tests. 



were able to organize program activities to attract and involve higher 
percentages of their faculties. This is most striking in attendance at 
staff development activities (designed to address the challenge dimension) 
and individual teacher implementation of strategieij. Both of these are 
recognized as critical factors in improving the performance of the school. 
No significant differences were found for the two program-structured 
activities (complete the DoES and discuss the results). 

Additional analyses were conducted to examine the relationship between 
performance on the DoES and school personnel's ratings of the conduct of the 
program in their schools (again obtained from responses to the School 
Survey ) . Mann-Whitney U tests were performed for all 20 school conduct 
indicators. As before, the top group of schools received consistently 
higher ratings than did the bottom group of schools (see Table 7 on the next 
page); 13 of the 20 produced significant findings in favor of the top group. 
In other words, personnel from the top group rated their schools* conduct of 
the program more positively than did personnel from the bottom group. Of 
most importance was faculty ratings of the importance of the challenge 
dimension, principal support of the school improvement effort, staff 
ownership, and availability of resources. 



Discussion 

The original impetus for conducting this study was to test the 
hypothesis that the magnitude of school improvement (as measured by gains on 
the challenge^ dimension) would correlate positively with faculty 
participation in the school improvement program. More simply, schools that 
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Table 7 

School Personnel Ratings of Program Conduct in Their Schools 



Indicator 


School 
D 


School 
B 


School 
A 


School 
£ 


School 
F 


School 
C 


b 

P 


Challenge dimension important 


4*71 


4.62 


4.56 


4.52 


4.46 


3.79 


<.05 


Considered several alternatives 


3»81 


3.62 


4.63 


4.21 


3.55 


4.07 


NS 


Strategies relevant 


4*59 


4.62 


4.63 


4.55 


3.77 


4.14 


<.05 


Enjoyed participation 


4.00 


4.23 


4.69 


4.42 


3.22 


3.74 


NS 


Worth the work 


4.12 


4.38 


4.5fi 


4.39 


3.67 


3.29 


NS 


Knew more afterwards 


4.71 


4.54 


4.56 


3.36 


3.46 


3.57 


<.05 


Clear differences now 


4.80 


4.31 


4.56 


4.42 


3.44 


3.21 


<.10 


Improved the school 


4.53 


4.23 


4.63 


4.30 


3.54 


3.36 


<.10 


Principal valued program 


4.94 


4.86 


4.81 


4.59 


3.90 


4.45 


<.05 


Principal supported program 


5.00 


4.92 


4.78 


4.68 


3.68 


4.67 


<.05 


Principal expected implementation 


4.88 


4.92 


4.66 


4.42 


4.18 


4.00 


4.05 


Shared leadership 


4.82 


4.48 


4.06 


4.32 


3.60 


4.00 


<.10 


Program effectiveness assessed 


4.56 


4.48 


4.00 


4.27 


3.40 


4.42 


NS 


Staff understood program 


4.76 


4.47 


4.07 


4.12 


3.84 


4.08 


NS 


When asked, all participated 


4.76 


4.53 


4.26 


4.27 


4.00 


4.38 


NS 


2/3 involved 


4.81 


4.84 


4.35 


4.12 


3.72 


4.50 


<.10 


Staff ownership 


4.13 


4.00 


3.76 


3.74 


2.95 


3.50 


-^.05 


Harmony about program 


4.65 


4.23 


4.28 


4.38 


3.38 


4.67 


NS 


Roles clear 


4.59 


4.23 


4.12 


4.15 


3.38 


4.00 


<.10 


Necessary resources allocated 


4.59 


4.35 


4.20 


3.82 


3.67 


4.00 


<.05 



*Mean ratings can range from a high of 5.00 (strongly agree) to a low of 1.00 (strongly disagree). 
^Probabilities are based on one- tailed Mann-Whitney U tests. 
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involve more staff in more program activities would show greater 
improvement. No support was found for this hypothesis. No consistent 
relationship was found between program participation and subsequent school 
improvement gains. However, the sample size for this study was extremely 
small and examined program implementation for a relatively short period of 
time. Additional research is needed in order to test this hypothesis more 
fully. 

Instead, the results prove the old adage, "the good just keep getting 
better. In this study, the schools that started ahead in the race for 
elementary excellence stayed ahead. They were able to use the program more 
effectively, involve more staff in program activities, develop more relevant 
strategies to address their schools* challenge dimension, and establish 
leadership and support for the program. These actions translated into 
higher gain scores on their respective challenge dimensions. 

Five of the six schools were able to use the program to make 
significant improvements in their status on the chosen challenge dimension. 
However, the bottom three schools were unable to make up initial starting 
differences between them and the top three schools. These findings are 
.similar to Chapter 1 findings in which interventions implemented 
consistently across all students (i.e.. Chapter 1 and non-Chapter 1) do not 
eliminate differences between the two groups. In order to reduce the gap, 
appropriate interventions must be designed that address the differential 
performance among students (Slavin, 1987). This rule also applies to school 
improvement efforts, especially those that seek to reduce the disparity 
among schools in a district. 

13 
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The message is very simple. School improvement efforts must pay 
attention to the context of the school. In this case, the context was 
reflected in the pretest performance of the individual schools on the DoES . 
Much of the second and third generation school effectiveness /improvement 
literature has cautioned educators not to forget this message (Corbett, 
Dawson, & Firestone, 1984; Lieberman, 1986; McLaughlin, 1978; Purkey & Smith, 
1983). However, up to now, few have reported data to back this point. 
Although this study involved a very small sample, it does provide. some 
empirical support. Good schools will be able to take most programs, run 
with them, and show improvement. Other schools may be able to use these 
same programs, but require additional assistance if these programs are going 
to have the same pay-off. This messege should not be forgotten in the 
design and conduct of school improvement efforts. 
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